Skip to content

Fix Windows CI flaps - #1145

Closed
mtmk wants to merge 2 commits into
release/3.0from
test/fix-windows-flaps
Closed

Fix Windows CI flaps#1145
mtmk wants to merge 2 commits into
release/3.0from
test/fix-windows-flaps

Conversation

@mtmk

@mtmk mtmk commented May 14, 2026

Copy link
Copy Markdown
Member

Test-only fixes for the Windows non-net8.0 flaps tracked in #1142. Targets release/3.0 for verification; cherry-pick to main after merge.

The main lever is tests/xunit.runsettings: MaxParallelThreads 4.0x -> 1.0x. With 4x parallelism on 4-core Windows runners, 16 tests run at once, each constructing a NatsConnection whose NatsReadProtocolProcessor.ReadLoopAsync is started via Task.Run. Many simultaneous mock-server tests starved the thread pool, so the read loop didn't get scheduled and SetupReaderWriterAsync timed out waiting for INFO. Serial execution removes the contention.

Also bumped ConnectTimeout to 30s on the six previously flapping tests, NatsUtils.ConnectRetryAsync budget 30s -> 60s, and widened TlsPreferTest's fallback to accept TimeoutException as evidence of a TLS attempt. Defense in depth; helpful if parallelism ever creeps back up.

Trade-off: Windows CI per job is ~13-19 min, up from ~7 min.

Verified: 12 of 12 Windows jobs green across 3 runs.

Fixes #1142.

  • Cherry-pick to main after merge

Default ConnectTimeout (2s) is too tight on contended Windows runners. Bump to 30s in the tests that have flapped on Windows non-net8.0 (PingCancellationTest, Utf8SubjectMockServerTest, ProtocolParserSizeCheckTest, TlsPreferTest, PinnedClientTest.Pin_id_from_headers, CancellationTokenTests.NextAsync). Widen the TlsPreferTest exception fallback to also accept TimeoutException as evidence of a TLS attempt (otherwise a stalled handshake under contention slips through). Bump NatsUtils.ConnectRetryAsync overall budget 30s -> 60s for the same reason.
@synadia-claude-reviewer

synadia-claude-reviewer Bot commented May 14, 2026

Copy link
Copy Markdown
Contributor

Claude finished @mtmk's task in 2m 46s —— View job


PR Review

  • Gather context (CLAUDE.md, PR diff)
  • Review changed files
  • Post inline comments for issues found
  • Write review summary

Test-only, no production source changes. Changes are correct and well-targeted.

TlsPreferTestConnectTimeout vs outer CTS interaction

The outer CancellationTokenSource is 10 s (cts), but ConnectTimeout is now 30 s. ConnectAsync() is called without cts.Token, so if the TLS handshake stalls the client could block for up to 30 s even after the 10 s CTS fires. In practice the CTS cancels stream.ReadAsync on the server side, which resets the TCP socket and unblocks ConnectAsync with an IOException well before 30 s. The 30 s is effectively a last-resort backstop for platforms (Windows) where socket cleanup is delayed. This is fine; the test can't hang indefinitely.

TimeoutException as TLS evidence

Adding TimeoutException to the tlsRelated check is a sound third fallback: on a stalled TLS handshake the ConnectTimeout fires, producing a TimeoutException in the exception chain. A plaintext path would complete well within the timeout, so a timeout is meaningful evidence TLS was attempted. See inline comment for a minor edge-case caveat (negligible in practice).

ConnectRetryAsync budget (30 s → 60 s) with per-connection ConnectTimeout = 30 s

Tests that now pass ConnectTimeout = TimeSpan.FromSeconds(30) and call ConnectRetryAsync() (60 s budget, 1 s inter-attempt delay) effectively get 1–2 connection attempts. This is intentional — the mock servers are on loopback and should connect on the first attempt; the generous timeout just avoids spurious failures on contended runners.

Overall

No correctness, security, or async-safety issues. The clientSentConnect.Should().BeFalse() guard remains the authoritative check against the underlying bug; the timeout widening and TimeoutException addition are reliable CI-stability improvements without weakening that guard.

// A connect-timeout while the client was already past the INFO read is also evidence the
// TLS path was attempted: the plaintext path would have completed the handshake within the
// timeout, while a stalled TLS handshake under runner contention shows up as a timeout.
var tlsRelated = causes.Any(c => c is AuthenticationException || c is SocketException || c is IOException || c is TimeoutException);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The TimeoutException addition is a reasonable fallback, but carries a small theoretical false-positive: if the client times out before receiving INFO (e.g. the fake server is unexpectedly slow to accept the TCP connection), neither clientAttemptedTls nor clientSentConnect will be set, yet TimeoutException would satisfy this check and the test passes without ever having verified the TLS upgrade path.

In practice the risk is negligible because the server is on loopback and writes INFO immediately. But it's worth noting that clientSentConnect.Should().BeFalse() is the authoritative bug-guard here — this tlsRelated check is belt-and-suspenders secondary evidence.

No change needed; just flagging the reasoning so future readers understand the trade-off.

PR #1145 run 1 showed PingCancellation/Utf8Subject mock-server tests still hanging 75s+ even with bumped ConnectTimeout. Stack always at SetupReaderWriterAsync waiting for INFO — the NatsReadProtocolProcessor's Task.Run never gets a thread. Under 4.0x parallelism we run 16 tests at once on 4-core Windows runners; many doing TCP mock-servers at the same time starves the thread pool. Drop to 1.0x to remove the contention.
@mtmk mtmk changed the title Fix Windows CI flaps by widening test connect timeouts Fix Windows CI flaps May 15, 2026
@mtmk mtmk mentioned this pull request May 15, 2026
@mtmk

mtmk commented May 15, 2026

Copy link
Copy Markdown
Member Author

covered by #1147

@mtmk mtmk closed this May 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant